Method and apparatus for interacting with television screen

ABSTRACT

A method of interacting with a television screen using a remote control having a viewing screen. The method includes the following: (1) displaying a boundary-identifier, on the viewing screen that is substantially transparent, to specify the boundary of an effective input-area; (2) detecting one or more positions being touched on the viewing screen after the boundary-identifier is displayed on the viewing screen; and (3) determining at least one mapped position on the television screen, wherein the at least one mapped position is mapped from a position among the one or more touching positions on the viewing screen of the remote control under a mapping operative to map one of an irregular quadrilateral and a trapezoid to a rectangular. The method also includes determining the effective input-area, which includes analyzing the image of the television screen obtained with a camera on the remote control.

RELATED APPLICATIONS

The present application is a continuation of patent application Ser. No.13/840,320, to be issued as U.S. Pat. No. 8,773,591, which is related toand claims benefit under 35 U.S.C. §119(e) from U.S. Provisional PatentApplication Ser. No. 61/682,759, filed Aug. 13, 2012, titled “METHOD ANDAPPARATUS FOR CONTROLLING A TELEVISION.” The present application is alsorelated to U.S. Pat. No. 8,736,773.

BACKGROUND

The present disclosure relates generally to controlling displayedcontents on the television screen at a visual distance.

FIG. 1 shows that multiple screens are often used to display the samegraphic content. Such multiple screens experience is considered by manyas the future of home computing or electronic gaming system. Forexample, the same graphic content can be simultaneously displayed on thescreen 200 of a television 60 and on the screen 72 of another portabledevice 70, such as, a table computer or a smartphone. Specifically,graphic objects 63 and 67 the screen 200 of the television 60 can besimultaneously displayed as graphic objects 73 and 77 respectively onthe screen 72 of the portable device 70. The screen 72 of the portabledevice 70 is often a touch-screen, and a user can use such touch-screento control the graphic contents on the television screen 200 or tointeract with the television 60 remotely. The portable device 70 oftenincludes a transceiver 79 to communicate wirelessly with the television60.

Applicant discovered that there is a need for an improved method andapparatus for controlling a graphic display at a distance. Specifically,despite the fact that the multiple screen system as shown in FIG. 1 iscommonly used by many people, older people with limited vision can findsuch multiple screen system is difficult to use. Some older people needa reading glass to read the contents on the screen 72 of the portabledevice 70, but they will have to take-off the reading glass to read thecontents on the screen 200 of the television 60. On the other hand, someother older people need a distance glass to read the contents on thescreen 200 of the television 60, but they will have to take-off thedistance glass to read the contents on the screen 72 of the portabledevice 70. It is desirable to control the displayed contents on thetelevision screen 200 or to interact with the television 60 remotelywithout the need to look at the same graphic contents on another screen(e.g., the screen 72 of the portable device 70).

BRIEF DESCRIPTION OF THE FIGURES

The accompanying figures, where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, together with the detailed description below, are incorporated inand form part of the specification, and serve to further illustrateembodiments of concepts that include the claimed invention, and explainvarious principles and advantages of those embodiments.

FIG. 1 shows that the same graphic content can be simultaneouslydisplayed on the screen of a television and on the screen of anotherportable device.

FIGS. 2A-2B depict that a user can use a transparent viewing screen of aremote control device to interact with the television remotely inaccordance with some embodiments.

FIGS. 3A-3B depict implementations for displaying theboundary-identifier on the transparent viewing screen of a remotecontrol device

FIG. 4 depict that, in accordance with some embodiments, a user cangenerate a set of touching positions on a sensing screen to determinethe boundary-identifier to be displayed.

FIG. 5 depict that, in accordance with some embodiments, a user cangenerate a set of touching positions on a variable-reflectivity screento determine the boundary-identifier to be displayed.

FIG. 6 depict that, in accordance with some embodiments, four cameras onthe television can be used to capture images of an eye though thetransparent viewing screen and to determine the mapping between aposition on the television screen to a corresponding position on theviewing screen.

FIG. 7 shows that the mapping between a position on the televisionscreen and a corresponding position on the viewing screen in accordancewith some embodiments.

FIG. 8 shows that, in accordance with some embodiments, a touch positioncan be mapped to a corresponding position on the television screen usingthe mapping determined in real time while the boundary-identifier forsuch mapping is displayed.

FIG. 9 shows that, in accordance with some embodiments, a touch positioncan be mapped to a corresponding position on the television screen usingthe mapping determined in real time with no boundary-identifierdisplayed.

FIG. 10 shows that, in accordance with some embodiments, the shape ofthe boundary-identifier can dynamically depend upon the location of theviewing screen and the surface orientation of the viewing screen.

FIG. 11 shows that, in accordance with some embodiments, the shape ofthe boundary-identifier can be configured to change with the surfaceorientation of the viewing screen.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions of some of the elements inthe figures may be exaggerated relative to other elements to help toimprove understanding of embodiments of the present invention.

The apparatus and method components have been represented whereappropriate by conventional symbols in the drawings, showing only thosespecific details that are pertinent to understanding the embodiments ofthe present invention so as not to obscure the disclosure with detailsthat will be readily apparent to those of ordinary skill in the arthaving the benefit of the description herein.

DETAILED DESCRIPTION

FIGS. 2A-2B depict that a user can use a transparent viewing screen 100of a remote control device 80 to control the graphic contents ontelevision screen 200 or to interact with the television 60 remotely,without the need to look at the same graphic contents on another screen,in accordance with some embodiments. In FIGS. 2A-2B, the transparentviewing screen 100 is operative to detect at least one touching position150 on the viewing screen 100 touched by one or more fingers of a user.In some implementations, the transparent viewing screen 100 can be atouch-screen. When the transparent viewing screen 100 is a touch-screenor a proximity-sensing screen, the at least one touching position 150touched by one or more fingers can be detected with electronics on theremote control device 80. In still other implementations, thetransparent viewing screen 100 can be a variable-reflectivity screen,and optical reflectivity at the touching position 150 on the screentouched by one or more fingers is implemented to change with touching.Such change of optical reflectivity due to finger touching can beobserved by one or more cameras fixed relative to the television screen200.

Additionally, the transparent viewing screen 100 is also operative todisplay a boundary-identifier 160 on the viewing screen to specify theboundary of an effective input-area 170 that is in the shape of aquadrilateral. In some implementations, the transparent viewing screen100 can be a transparent LCD display or a transparent OLED display. Theboundary-identifier 160 can be displayed with four straight lines 161,162, 163, and 164 forming the four sides of a quadrilateral. In someimplementations, the transparent viewing screen 100 may only need todisplay binary levels to show the four straight lines 161, 162, 163, and164. Certainly, in some implementations, the transparent viewing screen100 with gray levels is used.

The remote control device 80 also includes a controller configured todetermine a mapped position 250 on the screen 200 that is mapped from acorresponding touching position 150 on the viewing screen 100 detectedby the viewing screen after the boundary-identifier 160 is displayed.The mapped position 250 is mapped from the corresponding touchingposition 150 with a mapping operative to map the quadrilateral 160 to adisplay boundary of the screen 200. The display boundary of the screen200 generally is in the shape of rectangular that is different from theshape of the quadrilateral 160. Unless the viewing screen 100 at someparticular location and is orientated in some particular direction, theshape of the quadrilateral 160 generally is not a rectangle, and it canbe in the shape of a trapezoid or an irregular quadrilateral. The remotecontrol device 80 can be implemented to wirelessly communicate with thetelevision 60. For example, as shown in FIG. 2A, the remote controldevice 80 can include a transceiver 89 to communicate with thetelevision 60 wirelessly.

In operation, after the boundary-identifier 160 is displayed on theviewing screen 100 to specify the boundary of the effective input-area170, a user can hold the remote control device 80 in front of thetelevision 60 and look at the screen 200 through the transparent viewingscreen 100 with one eye located at the position 90. Then, as shown inFIG. 2B, the user can adjust the location and/or the orientation of theviewing screen 100 to align the boundary of the effective input-area 170with the display boundary of the screen 200. The user can also shift theposition 90 of the eye to make the alignment. Once the boundary of theeffective input-area 170 is aligned with the display boundary of thescreen 200, the user can use the effective input-area 170 as the proxytouch surface for the screen 200, because each touch point within theeffective input-area 170 has a one-to-one corresponding relationshipwith one equivalent touch point on the screen 200.

In the implementations as shown in FIGS. 2A-2B, the boundary-identifier160 is displayed as four straight lines 161, 162, 163, and 164 formingthe four sides of a quadrilateral. There are also other implementationsfor displaying the boundary-identifier 160. For example, in oneimplementation as shown in FIG. 3A, the boundary-identifier 160 isdisplayed as four corner points 101, 102, 103, and 104 specifying thecorners of a quadrilateral that defines the effective input-area 170. Inanother implementation as shown in FIG. 3B, areas outside thequadrilateral for defining the effective input-area 170 are changed toopaque or semi-opaque to function as the boundary-identifier 160, withthe edges of the effective input-area 170 clearly defined.

There are different ways of determining the effective input-area 170 andthe boundary-identifier 160 on the viewing screen 100 before the correctboundary-identifier 160 is displayed on the viewing screen. In oneimplementation, as shown in FIG. 4, the user can look at the screen 200through the transparent viewing screen 100 with one eye located at theposition 90 while keeping the viewing screen 100 steadily at aparticular location and orientation, and place a finger on the viewingscreen 100 to trace the expected boundary of the effective input-area170. During the tracing, a set of touching positions on the viewingscreen 100 is detected. Subsequently, the boundary of the effectiveinput-area 170 can be determined from this set of touching positions,and the boundary-identifier 160 on the viewing screen 100 can bedetermined as well. Instead of tracing the expected boundary of theeffective input-area 170, in another implementation, the expected fourcorner points 101, 102, 103, and 104 of the effective input-area 170 aretouched by a user to generate a set of touching positions, and this setof touching positions can be used to determine the boundary-identifier160.

In some implementations, when the viewing screen 100 is implemented as asensing screen (e.g., a touch-screen or a proximity-sensing screen), theset of touching positions can be determined by the electronics on theremote control 80. In some other implementations, as shown in FIG. 5,when the transparent viewing screen 100 is implemented as avariable-reflectivity screen, the set of touching positions can bedetermined by one or more cameras 310 fixed relative to the televisionscreen 200. The reflectivity of the view screen at the position pressedby a finger can change optical reflectivity at certain wavelength. Insome implementations, the position on the view screen pressed by afinger may appear to have its gray level changed or have its colorchanged. The change of gray level changed or the change of color can beobserved by the camera 310. In the implementations shown in FIG. 5, theone or more cameras 310 can be implemented on the frame of thetelevision 60 itself. In other implementations, the one or more cameras310 can implemented on a separated box that is fixed relative to thetelevision screen 200 during operation, and such separated box can bemoved relative to the television 60 when it is not in used duringoperation. In FIG. 4 and FIG. 5, once the effective input-area 170 onthe viewing screen 100 is determined, the mapping that maps a position(X, Y) on the television screen 200 to a corresponding position (x, y)on the viewing screen 100 can be determined, and alternatively, themapping that maps a position (x, y) on the viewing screen 100 to acorresponding position (X, Y) on the television screen 200 can also bedetermined.

In addition to manually determine the effective input-area 170 on theviewing screen 100 as shown in FIG. 4 and FIG. 5, it is also possible toautomatically determine the effective input-area 170. As shown in FIG.6, four cameras 301, 302, 303, and 304 fixed relative to the televisionscreen 200 are used to capture images of the eye at position 90 thoughthe transparent viewing screen 100. With these captured images, themapping that maps a position (X, Y) on the television screen 200 to acorresponding position (x, y) on the viewing screen 100 can bedetermined, and alternatively, the mapping that maps a position (x, y)on the viewing screen 100 to a corresponding position (X, Y) on thetelevision screen 200 can also be determined. With these capturedimages, the effective input-area 170 on the viewing screen 100 can alsobe automatically determined. FIG. 7 illustrates the forward and reversemapping between a position (X, Y) of point 250 on the television screen200 to a corresponding position (x, y) of point 150 on the viewingscreen 100 in accordance with some embodiments. In some implementations,such forward mapping or reverse mapping can be determined by thecontroller in the television 60, and the determined mapping can bewirelessly communicated to the remote control 80.

In one implementation, each of the captured images includes the image ofthe eye at position 90 within the image of the rectangular boundary ofthe viewing screen 100. With these captured images, the positions of thefour cameras 301, 302, 303, and 304 in the coordinate X-Y-Z fixedrelative to the television screen 60 are mapped to the correspondingpositions in the coordinate x-y-z fixed relative to the viewing screen100. If the positions of the four cameras 301, 302, 303, and 304 in thecoordinate X-Y-Z fixed relative to the television screen 200 are knownto be at predetermined positions, and if the corresponding positions inthe coordinate x-y-z fixed relative to the viewing screen 100 aremeasured from the captured images, the mapping that maps a position (X,Y) on the television screen 200 to a corresponding position (x, y) onthe viewing screen 100 can be determined. Consequently, as shown in FIG.6, when the positions 201, 202, 203, and 204 are mapped to thecorresponding positions 101, 102, 103, and 104 on the viewing screen 100using the mapping determined from the captured images, the effectiveinput-area 170 can also be determined. In some implementations, thecorresponding positions 101, 102, 103, and 104 on the viewing screen 100can be determined by the controller in the television 60, and thedetermined corresponding positions can be wirelessly communicated to theremote control 80.

In some implementations, the positions 201, 202, 203, and 204 are thecorner positions of a display area on the television screen 200. Inother implementations, the positions 201, 202, 203, and 204 can be otherrecognizable positions fixed relative to the television screen 200, andsuch recognizable positions can be used for the alignment of the viewingscreen 100. Accordingly, when the positions 201, 202, 203, and 204 arenot the corner positions of a display area on the television screen 200,boundary-identifier 160 can be used to specify an area other than theeffective input-area 170.

FIG. 8 shows that, in accordance with some embodiments, a touch position(x, y) on the viewing screen can be mapped to a corresponding position(X, Y) on the television screen 200 using the mapping determined in realtime while the boundary-identifier 160 for such mapping is displayed inreal time. During operation, if the user aligns the boundary-identifier160 with the edges of the television screen 200, a touch position 150can be mapped to a corresponding position 250 on the television screen200 with a mapping that is known or can be determined.

In some implementations, as shown in FIG. 9, the boundary-identifier 160does not have to be displayed on the view screen 100, if the mappingbetween a corresponding position (x, y) on the viewing screen 100 andthe corresponding position (X, Y) on the television screen 200 can bedetermined in real time. In FIG. 9, a touch position 150 can be mappedto a corresponding position 250 on the television screen 200 using themapping determined in real time. With the embodiment as shown in FIG. 9,no boundary-identifier is displayed, because the user does not have touse a boundary-identifier on the viewing screen 100 to make alignmentwith some recognizable positions on the television screen 200 before atouch position 150 can be used as a proxy touching position for acorresponding position 250 on the television screen 200.

In some implementations, as shown in FIG. 10, the boundary-identifier160 on the viewing screen 100 can be configured to have a shape thatdynamically depends upon the location of the viewing screen 100 and thesurface orientation of the viewing screen 100. Here, the surfaceorientation of the viewing screen 100 is the orientation of the normalvector n that is perpendicular to the viewing screen 100. In someimplementations, the location of the viewing screen 100 can becharacterize by the position (X₀, Y₀, Z₀) of the origin 180 in thecoordinate x-y-n fixed relative to the viewing screen 10.

In some implementations, as shown in FIG. 11, even if theboundary-identifier 160 on the viewing screen 100 is only manuallydetermined with the mechanism as shown in FIG. 4 or FIG. 5, theboundary-identifier 160 on the viewing screen 100 can still beconfigured to have its shape dynamically depends upon the surfaceorientation of the viewing screen 100. For example, the remote control80 can have gyroscopes or accelerometers to determine the surfaceorientation of the viewing screen 100 or the change of the surfaceorientation.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen having asurface orientation thereof defined by the orientation of the normalvector perpendicular to the viewing screen. The method includes thefollowing: (1) detecting, with electronics on the remote control, afirst set of positions being touched on the viewing screen while theviewing screen is maintained at a first position and at a first surfaceorientation; (2) analyzing the set of positions being touched todetermine the boundary of an effective input-area that has a shapesubstantially matching the shape of a quadrilateral; (3) displaying aboundary-identifier, on the viewing screen that is substantiallytransparent, to specify the boundary of the effective input-area; (4)detecting one or more positions being touched on the viewing screenafter the boundary-identifier is displayed on the viewing screen; and(5) determining a mapping operative to map the quadrilateral to arectangular that has a shape different from the shape of saidquadrilateral. In some implementations, the quadrilateral can be anirregular quadrilateral, or a trapezoid.

The method can further include determining at least one mapped positionon the television screen, wherein the at least one mapped position ismapped from a position among the one or more touching positions on theviewing screen of the remote control under said mapping.

The method can further include determining the shape of the effectiveinput-area while the viewing screen is at a current surface orientation.In some implementations, such determining the shape of the effectiveinput-area can include measuring the first surface orientation of theviewing screen, and measuring the current surface orientation of theviewing screen. In some implementations, such determining the shape ofthe effective input-area can include analyzing multiple shape-settingparameters including (1) the shape of the effective input-area while theviewing screen is at the first surface orientation, (2) the firstsurface orientation of the viewing screen, and (3) the current surfaceorientation of the viewing screen.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen. The methodincludes the following: (1) displaying a boundary-identifier, on theviewing screen that is substantially transparent, to specify theboundary of an effective input-area; (2) detecting one or more positionsbeing touched on the viewing screen after the boundary-identifier isdisplayed on the viewing screen; and (3) determining at least one mappedposition on the television screen, wherein the at least one mappedposition is mapped from a position among the one or more touchingpositions on the viewing screen of the remote control under a mappingoperative to map one of an irregular quadrilateral and a trapezoid to arectangular. In some implementations, the method can include determiningthe mapping operative to map one of an irregular quadrilateral and atrapezoid to a rectangular. In some implementations, the method caninclude determining the shape of the effective input-area, and analyzingthe shape of the effective input-area to determine said mapping.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen. The methodincludes the following: (1) determining the shape of an effectiveinput-area under an operation condition that the shape of an effectiveinput-area depends upon at least one of the surface orientation of theviewing screen and the location of the viewing screen, wherein thesurface orientation of the viewing screen is the orientation of thenormal vector perpendicular to the viewing screen; (2) displaying aboundary-identifier, on the viewing screen that is substantiallytransparent, to specify the boundary of the effective input-area; and(3) detecting one or more positions being touched on the viewing screenafter the boundary-identifier is displayed on the viewing screen. Insuch method, said determining the shape of the effective input-areacomprises (1) determining a quadrilateral for mapping to a rectangularthat has a shape different from the shape of said quadrilateral and (2)matching the shape of effective input-area substantially with the shapeof said quadrilateral. In some implementations, the method can includedetermining a mapping operative to map said quadrilateral to saidrectangular.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen. The methodincludes the following: (1) displaying a boundary-identifier, on theviewing screen that is substantially transparent, to specify theboundary of an effective input-area having a shape thereof dynamicallydepending upon at least one of the location of the viewing screen andthe surface orientation of the viewing screen, wherein the surfaceorientation of the viewing screen is the orientation of the normalvector perpendicular to the viewing screen; and (2) detecting one ormore positions being touched on the viewing screen after theboundary-identifier is displayed on the viewing screen. In someimplementations, the effective input-area has a shape that isessentially a quadrilateral, wherein said quadrilateral includes one ofan irregular quadrilateral, a trapezoid, and a rectangular.

In some implementations, the method can further include determining amapping operative to map a quadrilateral to a rectangular that has ashape different from the shape of said quadrilateral. In someimplementations, the method can further include determining the shape ofthe effective input-area, and analyzing the shape of the effectiveinput-area to determine a mapping operative to map a quadrilateral to arectangular that has a shape different from the shape of saidquadrilateral.

In some implementations, said displaying a boundary-identifiercomprises: displaying the boundary-identifier to specify the boundary ofan effective input-area having the shape thereof dynamically dependingupon BOTH the location of the viewing screen and the surface orientationof the viewing screen. In some implementations, said displaying aboundary-identifier comprises: displaying the boundary-identifier tospecify the boundary of the effective input-area while substantiallymaintaining the shape of the effective input-area when the viewingscreen is rotated about an axis parallel to the normal vector of theviewing screen. In some implementations, said displaying aboundary-identifier comprises: displaying the boundary-identifier tospecify the boundary of an effective input-area having the shape thereofdynamically depending upon the surface orientation of the viewingscreen. In some implementations, said displaying a boundary-identifiercomprises: displaying the boundary-identifier to specify the boundary ofan effective input-area having the shape thereof dynamically dependingupon the location of the viewing screen. In some implementations, theeffective input-area also has a size thereof dynamically depending upona distance between the viewing screen and a reference looking-point.

In some implementations, the method can further include determining atleast one mapped position on the television screen, wherein the at leastone mapped position is mapped from a position among the one or moretouching positions on the viewing screen of the remote control under amapping operative to map a quadrilateral to a rectangular, wherein saidrectangular having a shape different from the shape of saidquadrilateral.

In some implementations, the method can further include determining atleast one mapped position on the television screen, wherein the at leastone mapped position is mapped from a position among the one or moretouching positions on the viewing screen of the remote control under amapping from the effective area on the viewing screen to the displayarea on the television screen under the constrain that the boundary ofthe effective area on the viewing screen is essentially mapped to theboundary of the display area on the television screen

With respect to each of the above described aspects of the invention,implementations of the invention can include one or more of thefollowing features.

The viewing screen can have a diagonal length between 40 mm to 400 mm.In some implementations, the viewing screen is a touching screen, andthe detecting the one or more positions being touched on the viewingscreen can include detecting the one or more positions being touched onthe viewing screen with electronics on the remote control.

The detecting the one or more positions being touched on the viewingscreen can include detecting the one or more positions being touched onthe viewing screen with one or more cameras fixed relative to thetelevision screen. In some implementations, the optical reflectivity atthe one or more positions being touched on the viewing screen can bechanged with touching.

In some implementations, the mapping is a mapping that is the reverse ofa forward mapping belonging to a mapping class, wherein a member mappingin said mapping class maps a position (X, Y) on the television screen toa position (x, y) on the viewing screen, and said member mapping isidentifiable by relationshipsx=x₀+[(X-X₀)C₁₁+(Y-Y₀)C₁₂)]/[(1+a(X-X₀)+b(Y-Y₀)] and y=y₀+[(X-X₀)C₂₁+(Y-Y₀) C₂₂)]/[(1+a(X-X₀)+b(Y-Y₀)] with parameters a, b, C₁₁, C₁₂,C₂₁, C₂₂, X₀, Y₀, x₀, and y₀. In some implementations, at least one ofthe parameters a and b is non-zero. In some implementations, X₀=0, Y₀=0,x₀=0, and y₀=0. Some of the methods can include determining the value ofthe parameters a, b, C₁₁, C₁₂, C₂₁, C₂₂, X₀, Y₀, x₀, and y₀ in saidmember mapping. Some of the methods can include (1) determining at leastfour positions on the viewing screen each corresponding to one of knownpositions on the television screen; and (2) applying said member mappingbetween each of the at least four positions on the viewing screen andthe corresponding known position on the television screen to determinethe value of the parameters a, b, C₁₁, C₁₂, C₂₁, and C₂₂ in said membermapping.

In some implementations, the mapping is a mapping belonging to a mappingclass wherein a member mapping maps a position (x, y) on the viewingscreen to a position (X, Y) on the television screen, and said membermapping is identifiable by relationshipsX=X₀+[(x-x₀)D₁₁+(y-y₀)D₁₂)]/[1+p(x-x₀)+q(y-y₀)]andY=Y₀+[(x-x₀)D₂₁+(y-y₀)D₂₂)]/[1+p(x-x₀)+q(y-y₀)] with parameters p, q,D₁₁, D₁₂, D₂₁, D₂₂, X₀, Y₀, X₀, and y₀. In some implementations, atleast one of the parameters p and q is non-zero. In someimplementations, X₀=0, Y₀=0, x₀=0, and y₀=0. Some of the methods caninclude determining the value of the parameters p, q, D₁₁, D₁₂, D₂₁,D₂₂, X₀, Y₀, x₀, and y₀ in said member mapping. Some of the methods caninclude (1) determining at least four positions on the viewing screeneach corresponding to one of known positions on the television screen;and (2) applying said member mapping between each of the at least fourpositions on the viewing screen and the corresponding known position onthe television screen to determine said member mapping.

The determining the mapping can include determining the mapping with acamera fixed relative to the television screen. The determining themapping can include determining the mapping with two cameras fixedrelative to the television screen. The determining the mapping caninclude determining the mapping with four cameras fixed relative to thetelevision screen. The determining the mapping can include determiningthe mapping with a camera on the remote control. The determining themapping can include determining the mapping with two cameras on theremote control.

The determining the shape of the effective input-area can includedetermining the shape with a camera fixed relative to the televisionscreen. The determining the shape of the effective input-area caninclude determining the shape with two cameras fixed relative to thetelevision screen. The determining the shape of the effective input-areacan include determining the shape with four cameras fixed relative tothe television screen. The determining the shape of the effectiveinput-area can include determining the shape with a camera on the remotecontrol. The determining the shape of the effective input-area caninclude determining the shape with two cameras on the remote control.

Some of the methods can include determining the position of the viewingscreen. Some of the methods can include determining the surfaceorientation of the viewing screen. Some of the methods can includedetermining both the surface orientation of the viewing screen and theframe orientation of the viewing screen. Some of the methods can includedetermining the frame orientation of the viewing screen.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen. The methodincludes the following: (1) displaying a boundary-identifier, on theviewing screen that is substantially transparent, to specify theboundary of the effective input-area; (2) detecting one or morepositions being touched on the viewing screen after theboundary-identifier is displayed on the viewing screen; and (3)determining at least one mapped position on the television screen,wherein the at least one mapped position is mapped from a position amongthe one or more touching positions on the viewing screen of the remotecontrol under a mapping that is the reverse of a forward mappingbelonging to a mapping class, wherein a member mapping in said mappingclass maps a position (X, Y) on the television screen to a position (x,y) on the viewing screen, and said member mapping is identifiable byrelationships x=x₀+[(X-X₀)C₁₁+(Y-Y₀)C₁₂)]/[(1+a(X-X₀)+b(Y-Y₀)] andy=y₀+[(X-X₀)C₂₁+(Y-Y₀)C₂₂)]/[(1+a(X-X₀)+b(Y-Y₀)] with parameters a, b,C₁₁, C₁₂, C₂₁, C₂₂, X₀, Y₀, x₀, and y₀, and wherein at least one of theparameters a and b is non-zero.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The remote control comprises a viewing screen. The methodincludes the following: (1) displaying a boundary-identifier, on theviewing screen that is substantially transparent, to specify theboundary of the effective input-area; (2) detecting one or morepositions being touched on the viewing screen after theboundary-identifier is displayed on the viewing screen; and (3)determining at least one mapped position on the television screen,wherein the at least one mapped position is mapped from a position amongthe one or more touching positions on the viewing screen of the remotecontrol under a mapping belonging to a mapping class wherein a membermapping maps a position (x, y) on the viewing screen to a position (X,Y) on the television screen, and said member mapping is identifiable byrelationships X=X₀+[(x-x₀)D₁₁+(y-y₀)D₁₂)]/[1+p(x-x₀)+q(y-y₀)] andY=Y₀+[(x-x₀)D₂₁+(y-y₀)D₂₂)]/[1+p(x-x₀)+q(y-y₀)] with parameters p, q,D₁₁, D₁₂, D₂₁, D₂₂, X₀, Y₀, x₀, and y₀, and wherein at least one of theparameters p and q is non-zero.

In one aspect, a method of interacting with a television screen using aremote control having a viewing screen in accordance with someembodiments is described in the following. The remote control comprisesa viewing screen having a surface orientation thereof defined by theorientation of the normal vector perpendicular to the viewing screen.The method includes the following: (1) imaging the viewing screen thatis substantially transparent with one or more cameras fixed relative tothe television screen while the viewing screen having aboundary-identifier displayed to specify the boundary of an effectiveinput-area that has a shape substantially matching the shape of aquadrilateral, wherein the quadrilateral includes one of an irregularquadrilateral and a trapezoid; and (2) detecting one or more positionsbeing touched on the viewing screen after the boundary-identifier isdisplayed on the viewing screen.

Implementations of the invention can include one or more of thefollowing features. The method can include determining a mappingoperative to map the quadrilateral to a rectangular that has a shapedifferent from the shape of said quadrilateral. The method can includedetermining at least one mapped position on the television screen,wherein the at least one mapped position is mapped from a position amongthe one or more touching positions on the viewing screen of the remotecontrol under said mapping. The method can include analyzing the shapeof the effective input-area in one or more images of the viewing screento determine a mapping operative to map the quadrilateral to arectangular that has a shape different from the shape of saidquadrilateral.

In some implementations, the method can further include the following:(1) detecting, with electronics on the remote control, a first set ofpositions being touched on the viewing screen while the viewing screenis maintained at a first position and at a first surface orientation;(2) analyzing the set of positions being touched to determine theboundary of the effective input-area. In other implementations, themethod can further include the following: (1) detecting, with at leastone of the one or more cameras fixed relative to the television screen,a first set of positions being touched on the viewing screen while theviewing screen is maintained at a first position and at a first surfaceorientation; and (2) analyzing the set of positions being touched todetermine the boundary of the effective input-area.

The method can include determining at least one mapped position on thetelevision screen, wherein the at least one mapped position is mappedfrom a position among the one or more touching positions on the viewingscreen of the remote control under a mapping operative to map one of anirregular quadrilateral and a trapezoid to a rectangular. The method canfurther include determining the mapping operative to map one of anirregular quadrilateral and a trapezoid to a rectangular. The method canfurther include analyzing the shape of the effective input-area in oneor more images of the viewing screen to determine said mapping.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The method includes the following: (1) imaging the viewingscreen that is substantially transparent with at least two cameras fixedrelative to the television screen, the viewing screen that issubstantially transparent; (2) detecting, with at least one of the atleast two cameras fixed relative to the television screen, one or morepositions being touched on the viewing screen; and (3) analyzing imagesof the viewing screen to determine the boundary of an effectiveinput-area that has a shape substantially matching the shape of aquadrilateral, wherein the quadrilateral includes one of an irregularquadrilateral and a trapezoid. In some Implementations, the viewingscreen is operative to display a boundary-identifier to specify theboundary of the effective input-area. In some implementations, themethod can include detecting, with the at least one of the at least twocameras fixed relative to the television screen, one or more positionsbeing touched on the viewing screen after a boundary-identifier isdisplayed on the viewing screen to specify the boundary of the effectiveinput-area.

In one aspect, a method of interacting with a television screen using aremote control in accordance with some embodiments is described in thefollowing. The method includes the following: (1) imaging the viewingscreen that is substantially transparent with at least two cameras fixedrelative to the television screen; (2) detecting, with at least one ofthe at least two cameras fixed relative to the television screen, one ormore positions being touched on the viewing screen; and (3) analyzingimages of the viewing screen to determine a mapping operative to map aquadrilateral to a rectangular that has a shape different from the shapeof said quadrilateral. In some implementations, said quadrilateral canbe one of an irregular quadrilateral and a trapezoid. In someImplementations, the viewing screen is operative to display aboundary-identifier to specify the boundary of the effective input-area.In some implementations, the method can include detecting, with the atleast one of the at least two cameras fixed relative to the televisionscreen, one or more positions being touched on the viewing screen aftera boundary-identifier is displayed on the viewing screen to specify theboundary of the effective input-area.

In one aspect, a remote control for controlling a television screen inaccordance with some embodiments is described in the following. Theremote control includes the following: (1) a viewing screen that issubstantially transparent and has a diagonal length between 40 mm to 400mm, wherein the viewing screen is operative to detect at least onetouching position on the viewing screen, and wherein the viewing screenis also operative to display a boundary-identifier on the viewing screento specify the boundary of an effective input-area; and (2) a controllerconfigured to determine the boundary of the effective input-area from afirst set of touching positions detected by the viewing screen while theviewing screen is maintained at a first position and at a first surfaceorientation screen, with the shape of the effective input-areasubstantially matching the shape of a quadrilateral, and to determine amapping operative to map the quadrilateral to a rectangular that has ashape different from the shape of said quadrilateral. In someimplementations, the controller is configured to determine one or moremapped touching position on the television screen, wherein each mappedtouching position on the television screen is mapped, under saidmapping, from a corresponding touching position on the viewing screendetected by the viewing screen after the boundary-identifier isdisplayed on the viewing screen. In some implementations, the remotecontrol also includes a transmitter configured to transmit to thetelevision wirelessly data describing said mapping.

In one aspect, a remote control for controlling a television screen inaccordance with some embodiments is described in the following. Theremote control includes the following: (1) a viewing screen that issubstantially transparent and has a diagonal length between 40 mm to 400mm, wherein the viewing screen is operative to detect at least onetouching position on the viewing screen, and wherein the viewing screenis also operative to display a boundary-identifier on the viewing screento specify the boundary of an effective input-area that has a shapesubstantially matching the shape of a quadrilateral; and (2) acontroller configured to determine one or more mapped touching positionon the television screen with a mapping operative to map thequadrilateral to a rectangular that has a shape different from the shapeof said quadrilateral, wherein each mapped touching position on thetelevision screen is mapped, under said mapping, from a correspondingtouching position on the viewing screen detected by the viewing screenafter the boundary-identifier is displayed on the viewing screen.

In one aspect, a remote control for controlling a television screen inaccordance with some embodiments is described in the following. Theremote control includes the following: (1) a viewing screen that issubstantially transparent and has a diagonal length between 40 mm to 400mm, wherein the viewing screen is operative to detect at least onetouching position on the viewing screen, and wherein the viewing screenis also operative to display a boundary-identifier to specify theboundary of an effective input-area on the viewing screen; (2)electronics configured to make the shape of the effective input-areachanging with at least one of the location of the viewing screen and thesurface orientation of the viewing screen, wherein the surfaceorientation of the viewing screen is the orientation of the normalvector of the viewing screen; and (3) a memory configured to store oneor more touching positions detected by the viewing screen after theboundary-identifier is displayed on the viewing screen.

In some implementations, said electronics configured to make the shapeof the effective input-area changing is further configured to make boththe size and the shape of the effective input-area substantiallyinvariant with respect to rotating of the viewing screen with respect tothe normal vector of the viewing screen. In some implementations, Theremote control can further include electronics configured to determineone or more mapped touching position on the television screen with amapping operative to map a quadrilateral to a rectangular that has ashape different from the shape of said quadrilateral, wherein eachmapped touching position on the television screen is mapped, under saidmapping, from a corresponding touching position on the viewing screendetected by the viewing screen after the boundary-identifier isdisplayed on the viewing screen, wherein the shape of the quadrilateralsubstantially matches the shape of the effective input-area.

Implementations of the invention can include one or more of thefollowing features. In some implementations, the remote control canfurther include a transmitter configured to transmit to the televisionwirelessly data describing the one or more touching positions detectedby the viewing screen. In some implementations, the remote control canfurther include a transmitter configured to transmit to the televisionwirelessly data describing the one or more mapped touching position.

In some implementations, said quadrilateral is one of an irregularquadrilateral and a trapezoid. In some implementations, saidquadrilateral is one of an irregular quadrilateral, a trapezoid, and arectangular. The remote control can further include a plurality ofgyroscopes. The remote control can further include a plurality ofaccelerometers.

In some implementations, the remote control can further include (1) acamera configured to obtain an image of the television screen, and (2)electronics configured for analyzing the image of the television screento determine the shape of the effective input-area. In someimplementations, the remote control can further include (1) two camerasconfigured to obtain images of the television screen, and (2)electronics configured for analyzing the images of the television screento determine the shape of the effective input-area.

In some implementations, the remote control can further include (1) acamera configured to obtain an image of the television screen, and (2)electronics configured for analyzing the image of the television screento determine said mapping. In some implementations, the remote controlcan further include (1) two cameras configured to obtain images of thetelevision screen, and (2) electronics configured for analyzing theimages of the television screen to determine said mapping.

In the foregoing specification, specific embodiments have beendescribed. However, one of ordinary skill in the art appreciates thatvarious modifications and changes can be made without departing from thescope of the invention as set forth in the claims below. For example,the television screen described above can be the display screen of atelevision, a video box, a game console, or a computer. The televisionscreen described above can also be an extended display of a mobiledevice, such as a smartphone or a tablet computer. Accordingly, thespecification and figures are to be regarded in an illustrative ratherthan a restrictive sense, and all such modifications are intended to beincluded within the scope of present teachings.

The benefits, advantages, solutions to problems, and any element(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeatures or elements of any or all the claims. The invention is definedsolely by the appended claims including any amendments made during thependency of this application and all equivalents of those claims asissued.

Moreover in this document, relational terms such as first and second,top and bottom, and the like may be used solely to distinguish oneentity or action from another entity or action without necessarilyrequiring or implying any actual such relationship or order between suchentities or actions. The terms “comprises,” “comprising,” “has”,“having,” “includes”, “including,” “contains”, “containing” or any othervariation thereof, are intended to cover a non-exclusive inclusion, suchthat a process, method, article, or apparatus that comprises, has,includes, contains a list of elements does not include only thoseelements but may include other elements not expressly listed or inherentto such process, method, article, or apparatus. An element proceeded by“comprises . . . a”, “has . . . a”, “includes . . . a”, “contains . . .a” does not, without more constraints, preclude the existence ofadditional identical elements in the process, method, article, orapparatus that comprises, has, includes, contains the element. The terms“a” and “an” are defined as one or more unless explicitly statedotherwise herein. The terms “substantially”, “essentially”,“approximately”, “about” or any other version thereof, are defined asbeing close to as understood by one of ordinary skill in the art, and inone non-limiting embodiment the term is defined to be within 10%, inanother embodiment within 5%, in another embodiment within 1% and inanother embodiment within 0.5%. The term “coupled” as used herein isdefined as connected, although not necessarily directly and notnecessarily mechanically. A device or structure that is “configured” ina certain way is configured in at least that way, but may also beconfigured in ways that are not listed.

It will be appreciated that some embodiments may be comprised of one ormore generic or specialized processors (or “processing devices”) such asmicroprocessors, digital signal processors, customized processors andfield programmable gate arrays (FPGAs) and unique stored programinstructions (including both software and firmware) that control the oneor more processors to implement, in conjunction with certainnon-processor circuits, some, most, or all of the functions of themethod and/or apparatus described herein. Alternatively, some or allfunctions could be implemented by a state machine that has no storedprogram instructions, or in one or more application specific integratedcircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic. Of course, acombination of the two approaches could be used.

Moreover, an embodiment can be implemented as a computer-readablestorage medium having computer readable code stored thereon forprogramming a computer (e.g., comprising a processor) to perform amethod as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, a CD-ROM, an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. Further, it is expected that one of ordinary skill,notwithstanding possibly significant effort and many design choicesmotivated by, for example, available time, current technology, andeconomic considerations, when guided by the concepts and principlesdisclosed herein will be readily capable of generating such softwareinstructions and programs and ICs with minimal experimentation.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separately claimed subject matter.

What is claimed is:
 1. A method of interacting with a television screenusing a remote control, wherein the remote control comprises a viewingscreen, the method comprising: determining an effective input-area onthe viewing screen that is substantially transparent, wherein saiddetermining includes analyzing the image of the television screenobtained with a camera on the remote control; displaying aboundary-identifier on the viewing screen to specify the boundary of theeffective input-area; detecting one or more positions being touched onthe viewing screen after the boundary-identifier is displayed on theviewing screen; determining at least one mapped position on thetelevision screen, wherein the at least one mapped position is mappedfrom a position among the one or more touching positions on the viewingscreen of the remote control under a mapping operative to map one of anirregular quadrilateral and a trapezoid to a rectangular.
 2. The methodof claim 1, further comprising: determining the mapping operative to mapone of an irregular quadrilateral and a trapezoid to a rectangular. 3.The method of claim 1, further comprising: analyzing the shape of theeffective input-area to determine said mapping.
 4. The method of claim1, wherein said detecting the one or more positions being touched on theviewing screen comprises: detecting the one or more positions beingtouched on the viewing screen with electronics on the remote control,wherein the viewing screen is a sensing screen.
 5. The method of claim1, wherein the viewing screen is a variable-reflectivity screen, withthe optical reflectivity changing at the one or more positions beingtouched on the viewing screen.
 6. The method of claim 1, wherein saiddetermining includes analyzing the images of the television screenobtained with two cameras on the remote control.
 7. The method of claim1, further comprising: determining the surface orientation of theviewing screen.
 8. The method of claim 1, further comprising:determining the frame orientation of the viewing screen.
 9. The methodof claim 1, further comprising: determining both the surface orientationof the viewing screen and the frame orientation of the viewing screen.10. A remote control for controlling a television screen comprising: aviewing screen that is substantially transparent and has a diagonallength between 40 mm to 400 mm, wherein the viewing screen is operativeto detect at least one touching position on the viewing screen, andwherein the viewing screen is also operative to display aboundary-identifier on the viewing screen to specify the boundary of aneffective input-area that has a shape substantially matching the shapeof a quadrilateral; a camera configured to obtain an image of thetelevision screen; and electronics configured for determining theeffective input-area that includes analyzing the image of the televisionscreen obtained with the camera.
 11. The remote control of claim 10,further comprising: two cameras configured to obtain images of thetelevision screen; and wherein the electronics is configured fordetermining the effective input-area that includes analyzing the imagesof the television screen obtained with the two cameras.
 12. The remotecontrol of claim 10, wherein said quadrilateral is one of an irregularquadrilateral and a trapezoid.
 13. The remote control of claim 10,further comprising: a transmitter configured to transmit to anotherdevice wirelessly data describing the one or more touching positionsdetected by the viewing screen.
 14. The remote control of claim 10,further comprising: a controller configured to determine one or moremapped touching position on the television screen with a mappingoperative to map the quadrilateral to a rectangular that has a shapedifferent from the shape of said quadrilateral, wherein each mappedtouching position on the television screen is mapped, under saidmapping, from a corresponding touching position on the viewing screendetected by the viewing screen after the boundary-identifier isdisplayed on the viewing screen.
 15. The remote control of claim 14,further comprising: a transmitter configured to transmit to anotherdevice wirelessly data describing the one or more mapped touchingposition.
 16. The remote control of claim 10, further comprising: acontroller configured to make the boundary-identifier displayed on theviewing screen to specify the boundary of the effective input-area andto make the shape of the quadrilateral changing with at least one of thelocation of the viewing screen and the surface orientation of theviewing screen while making both the size and the shape of thequadrilateral substantially invariant with respect to rotating of theviewing screen with respect to the normal vector perpendicular to theviewing screen, wherein the surface orientation of the viewing screen isthe orientation of the normal vector of the viewing screen.
 17. Theremote control of claim 10, further comprising: a plurality ofgyroscopes.
 18. The remote control of claim 10, further comprising: aplurality of accelerometers.
 19. A remote control for controlling atelevision screen comprising: a viewing screen that is substantiallytransparent and has a diagonal length between 40 mm to 400 mm, whereinthe viewing screen is operative to display a boundary-identifier on theviewing screen to specify the boundary of an effective input-area thathas a shape substantially matching the shape of a quadrilateral; acontroller configured to make the boundary-identifier displayed on theviewing screen to specify the boundary of the effective input-area andto make the shape of the quadrilateral changing with at least one of thelocation of the viewing screen and the surface orientation of theviewing screen while making both the size and the shape of thequadrilateral substantially invariant with respect to rotating of theviewing screen with respect to the normal vector perpendicular to theviewing screen, wherein the surface orientation of the viewing screen isthe orientation of the normal vector of the viewing screen; a cameraconfigured to obtain an image of the television screen; and electronicsconfigured for determining the effective input-area that includesanalyzing the image of the television screen obtained with the camera20. The remote control of claim 19, further comprising: two camerasconfigured to obtain images of the television screen; and wherein theelectronics is configured for determining the effective input-area thatincludes analyzing the images of the television screen obtained with thetwo cameras.